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ABSTRACT 

This  report  describes  a  project  which  developed  an  objective 
measurement  of  motivation  using  Flanagan's  critical  incident 
technique.   The  sample  population  was  U.  S.  Navy  Jezebel  operators. 
The  report  includes  a  rating  form  which  was  used,  the  statistical 
computations  of  the  data  recorded  and  a  new,  modified  form.  The 
rating  form  was  found  to  have  a  high  interrater  reliability;  many 
of  the  scores  on  the  items  had  an  exceptionally  high  correlation 
with  the  total  score  received  and  the  study  generated  a  particularly 
good  operational  definition  of  motivation. 
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I.   INTRODUCTION 

The  problem  of  increasing  productivity  or  improving  performance 
is  one  which  concerns  everyone  in  a  managerial  position.  Numerous 
studies  have  been  conducted  to  identify  and  attempt  to  control  as 
many  of  the  factors  that  have  a  bearing  on  output  as  possible.  One 
of  the  best  descriptions  of  the  interrelated  nature  of  the  many 
elements  involved  can  be  found  in  Sutermeister  (Sutermeister,  1969) . 
In  his  conceptual  scheme  the  factors  most  closely  related  and 
directly  influential  on  productivity  are  the  physical  elements  of 
the  job  (i.e.,  technological  development,  raw  materials,  job  layout 
and  methods)  and  the  employee's  job  performance  which  he  attributes 
to  a  complex  interaction  of  ability  and  motivation.  While  ability 
is  not  a  precisely  defined  or  completely  understood  concept  it  is 
measurable  and  reliably  modifiable.   The  motivational  level  of 
employees  is  a  different  matter  altogether,  both  in  terms  of  the 
degree  of  agreement  on  the  elements  of  motivation  and  in  the 
untapped  potential  for  greater  understanding  and  selective  modifi- 
cation of  motivation.   It  can  safely  be  said  that  there  is  virtually 
universal  acceptance  of  the  important  influence  that  motivation  has 
in  a  broad  range  of  human  endeavor. 

In  the  past  25  years  there  has  been  a  proliferation  of  theories 
of  motivation.  Many  of  these,  such  as  Maslow's  "Need  Hierarchy 
Theory,"  (Mas low,  1943) , "McGregor's  Theory  Y"  (McGregor,  1960), 
Victor  Vroom's  "Instrumentality  Theory"  (Vroom,  1964)  and  Herzberg's 


"Two  Factor  Theory"  (Herzberg,  1959)  have  generated  widespread 
interest.  More  recently  several  organizations  have  come  into 
existence  that  purport  to  selectively  modify  the  level  of  motivation 
of  individuals,  groups,  or  of  entire  organizations  (for  example, 
Success  Motivation  Institute  of  Waco,  Texas). 

There  is  one  fundamental  element  that  seems  to  be  missing  in 
both  the  theoretical  and  the  applied  approaches  to  motivation.  This 
missing  element  is  a  method  of  determining  the  level  of  motivation 
(as  represented  in  performance)  of  a  group  or  an  individual.  Such 
a  technique  is  necessary  to  enable  investigators  to  detect  differences 
or  changes  in  motivation  and  to  test  the  effects  of  incentives  or  of 
management  policy.  An  intensive  search  of  the  available  literature 
failed  to  locate  an  objective  instrument  for  the  measurement  of 
motivation. 

It  was  important  in  the  initial  stages  of  this  project  to 
limit  the  area  to  be  investigated  and  to  select  a  definition  of 
motivation  upon  which  to  base  the  study.   It  was  assumed  that 
motivation  requires  force  and  direction  and  the  study  was  limited 
to  the  work  context.  Motivation  then  is  task  related  effort  expended 
in  directed  activity.   In  the  investigation  of  the  literature  it 
became  increasingly  obvious  that  there  are  two  quite  distinct 
approaches  to  motivation,  one  is  the  classical  psychological  view 
of  motivation  (which  refers  to  the  processes  governing  choices  made 
among  alternative  forms  of  voluntary  activity),  the  other  is 
motivation  as  seen  by  the  management  theorists.   This  dichotomy  seems 
to  have  developed  from  the  method  of  approaching  the  problem.  The 
psychologists  have  looked  at  physiological  states,  generally  resulting 


from  deprivation,  and  have  inferred  motivation  from  observed 
behavior,  content  analysis,  or  stated  attitudes  whereas  researchers 
dealing  with  an  industrial  or  social  organization  have  tended  to 
view  motivation  as  the  element  of  performance  that  can  not  be 
explained  by  any  other  known  variable.  This  investigation  was 
oriented  along  the  latter  lines,  that  is  the  scope  was  limited  to 
the  world  of  work  and  the  motivation  to  perform  a  particular  task 
or  group  of  tasks . 


II.   NATURE  OF  THE  PROBLEM 

In  the  development  of  a  measurement  of  motivation,  the  major 
problem  was  to  determine  what  behavior  is  characteristic  of  a  highly- 
motivated  task-oriented  worker.   Specifically  it  was  necessary  to 
determine  whether  most  motivated  men  in  a  reasonably  limited  job 
category  can  be  expected  to  be  identifiable  through  a  few  acts  or 
a  few  classes  of  characteristic  action  and  whether  certain  actions 
are  characteristic  of  all  or  at  least  a  significant  percentage  of 
highly  motivated  people  doing  approximately  the  same  job.  The 
second  important  consideration  was  to  insure  that  a  reasonably 
accurate  performance  measure  was  available  for  the  population 
sampled.   Failure  to  consider  this  element  would  result  in  an 
instrument  which  could  not  be  "calibrated"  or  tested  in  a  real  world 
setting.  As  a  larger  percentage  of  jobs  have  a  less  direct  influence 
on  production,  this  requirement  becomes  much  more  difficult  to  satisfy 
Because  of  this,  it  was  necessary  to  conduct  the  study  outside  the 
setting  of  the  production  line  to  insure  relevance  to  modern  industry 
and  the  Armed  Forces . 

Whenever  a  measurement  instrument  purports  to  measure  a  concept 
as  ill-defined,  poorly  understood  and  with  the  many  dimensions  of 
motivation,  the  problems  of  validity  and  objectivity  become  quite 
severe.   Is  it  really  motivation  that  we  are  measuring  or  some 
complex  and  obscure  artifact?  Do  the  raters  see  the  behavior  of 
the  people  they  are  rating  as  demonstrative  of  an  element  of  a 


complex  characteristic?  How  competent  are  raters  at  generalization 
from  one  specific  act  to  a  global  evaluation?  How  much  is  the 
evaluation  of  motivation  influenced  by  performance?  Indeed,  how  much 
can  performance  and  motivation  be  separated? 

As  a  result  of  these  questions  it  was  imperative  to  select  an 
experimental  technique  to  minimize  the  influence  of  unwanted  variables. 
A  unique  opportunity  and  certain  necessary  encouragement  to  study  this 
problem  were  presented  by  Dr.  John  P.  Smith  of  the  United  States  Naval 
Personnel  and  Training  Research  Laboratory,  San  Diego,  California.  He 
and  his  Jezebel  Study  Group  had  developed  an  effective  performance 
measure  for  operator  proficiency  and  had  done  some  work  with  a 
preliminary  motivation  rating  scheme.  As  a  part  of  the  field  testing 
of  their  proficiency  test  a  set  of  general  demographic  information 
was  gathered  about  each  man.  This  included  both  biographical  and 
experience  data.  As  an  element  of  this  information,  supervisors 
were  asked  to  rate  the  men  working  for  them  on  their  degree  of 
motivation.  The  instructions  for  this  rating  included  the  comment 
that  "motivation  implies  consistent  willingness  to  give  effort  to  a 
task  or  duty  even  in  the  absence  of  demand  by  a  military  superior;  it 
implies  a  belief  in  the  importance  of  the  Air  ASW  mission  and  it 
implies  a  constructive  attitude  toward  other  squadron  personnel." 
The  men  were  given  a  numerical  rating  from  one  to  seven,  this  score 
was  used  for  analysis  and  tests  of  correlation  with  the  performance 
rating  and  the  other  factors.   Their  analysis  revealed  that  two  of 
the  factors  (rank  and  motivation)  had  a  significantly  higher 


correlation  with  the  performance  rating  than  did  the  other  elements 

and  this  contribution  was  not  predicted  by  any  of  the  aptitude  or 

other  personal  characteristics  such  as  GCT,  ARI,  age  or  education. 

The  following  paragraph  is  an  excerpt  from  their  report: 

The  guidelines  given  to  the  raters  to  define  motivation 
are  consistent  with  characteristics  which  would  be  expected  to 
be  present  in  career  personnel;  if  so,  rank  and  motivation 
rating  should  be  correlated.   Investigation  of  this  point 
resulted  in  a  moderately  low  Pearsonian  correlation  coefficient 
of  .36.   The  two  variables  are  related,  but  they  are  also 
sufficiently  independent  of  each  other  to  strongly  suggest 
that  further  study  of  this  approach  to  motivation  may  be 
profitable.   Further  data  analysis  has  been  carried  on  to 
evaluate  more  precisely  the  significance  of  the  motivation 
ratings  using  analysis  of  coveriance  and  multiple  correlation. 

(Naval  Personnel  and  Training  Research  Laboratory  Technical  Bulletin 

STB  11-5.) 
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III.   EXPERIMENTAL  PROCEDURE 

A.   GENERAL  PROCEDURE 

In  the  previous  section  a  number  of  the  problems  inherent  in 
developing  an  instrument  for  the  measurement  of  motivation  were 
discussed.  A  constant  concern  in  the  selection  of  an  appropriate 
procedure  and  in  processing  the  data  that  were  generated  was  to 
minimize  the  influence  of  these  problems.   In  keeping  with  this 
philosophy,  the  method  of  choice  was  John  C.  Flannigan's  Critical 
Incident  Technique.  Very  briefly  the  Critical  Incident  Technique 
involves  describing  a  situation  for  the  person  being  interviewed 
and  asking  him  to  relate  an  incident  that  fits  this  situation. 
Specifically,  in  this  project  the  men  were  asked  to,  "Think  of 
the  most  highly  motivated  Jezebel  operator  that  you  know.  What  is 
it  about  him  that  caused  you  to  select  him?  Please  be  as  specific 
as  you  can."  After  asking  this  question  the  only  additional 
comments  that  were  made  by  the  interviewer  were  requests  for  greater 
amplification  of  the  incidents  that  were  reported  or  for  more  specific 
incidents  (Flannigan,  1954) . 

Interviews  were  conducted  on  two  occasions.  The  first  set  of 
interviews  were  with  one  senior  petty  officer  from  each  of  four  VP 
(Patrol)  squadrons  and  two  petty  officers  from  another  squadron. 
Since  this  set  represented  a  horizontal  section  across  the  rank 
structure  the  second  interviews  involved  a  vertical  section  of 
eight  operators  whose  paygrade  varied  from  E3  to  E6  (AWAN-AW1)  in 
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a  single  squadron.   It  was  felt  that  this  procedure  would  tend  to 
minimize  the  possibility  that  a  significant  difference  existed 
between  the  conception  of  motivated  behavior  held  by  junior  and 
senior  petty  officers.  The  initial  intent  was  to  investigate  the 
possibility  of  developing  incidents  descriptive  of  highly  motivated 
and  poorly  motivated  behavior.  This  was  unsuccessful  with  the 
interview  technique  that  was  used;  only  positively  motivated  behavior 
was  described  by  specific  incidents.  There  are  several  possible 
reasons  for  this  and  they  are  described  in  the  Results  and  Discussion 
Section. 

A  total  of  97  incidents  were  collected  and  after  closely  related 
items  were  combined  or  eliminated  55  separate  items  were  retained 
for  use.   Several  independent  attempts  were  made  to  order  these 
items  into  dimensions  of  motivation  and  to  produce  a  scale  of 
increasingly  favorable  or  graded  desirability.   These  attempts 
were  unsuccessful;  the  incidents  did  not  naturally  fall  into 
distinct  categories  and  they  did  not  have  varying  degrees  of 
desirability.  Twenty  items  were  selected  for  inclusion  on  the 
rating  form.   The  basis  for  selection  was  the  frequency  that  a 
behavior  was  reported  or  the  particularly  characteristic  nature 
of  the  behavior.   It  is  important  to  note  that  in  several  interviews 
one  incident  was  sufficient  to  cause  a  man  to  be  singled  out  and 
described.   In  general  the  behavior  that  was  described  in  this 
manner  represented  a  considerable  amount  of  time  and  effort  on  a 
project  or  a  breakthrough  that  was  achieved  through  intense  interest 
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and  attention  to  the  airborne  ASW  problem.  An  example  of  the 
Motivation  Rating  Form  is  included  as  Appendix  A. 

In  March  1971,  the  form  was  tested  at  Naval  Air  Station,  Moffett 
Field.   The  squadron  (VP-19)  had  18  Jezebel  operators  in  the  AW 
division  and  five  senior  petty  officers  familiar  with  each  of  the 
operators  who  were  available  to  rate  the  men.   The  raters  were 
asked  to  indicate  when  they  were  basing  their  evaluations  on 
observation  or  an  expectation.   This  was  to  attempt  to  separate 
actually  observed  behavior  from  an  estimation  that  the  man  would 
reasonably  be  expected  to  perform  in  the  described  manner  in  an 
appropriate  situation.  After  completing  the  form,  each  rater  was 
asked  to  comment  on  his  general  attitude  about  the  form,  its 
acceptability,  completeness  and  any  specific  comments  about  the 
description  of  motivation  that  it  provided.   Information  from  these 
interviews  is  the  basis  of  part  of  the  Analysis  of  Data  section. 
Following  the  interview  each  rater  was  asked  to  rank  order  each  of 
the  men  according  to  their  mission-oriented  motivation.   They  were 
asked  to  minimize  the  influence  of  the  form  they  had  just  completed 
and  to  the  best  of  their  ability  eliminate  the  influence  of  on  the 
job  performance.   The  information  that  was  collected  was  analyzed 
using  several  statistical  techniques  that  are  described  in  the 
following  section. 

B.   SAMPLE  POPULATION 

The  population  that  was  sampled  in  this  project  was  drawn  from 
United  States  Navy  Jezebel  Operators.  Jezebel  is  a  system  for  the 
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airborne  classification  and  localization  of  submarines  using 
information  transmitted  from  sonobuoys .   Specific  details  about  the 
system  and  the  techniques  of  analysis  are  classified,  however,  they 
involve  a  combination  of  training,  experience,  memory,  mechanical 
and  electronic  devices  and  onboard  reference  publications. 

The  minimum  requirements  to  become  an  Antisubmarine  Warfare 
Operator  (AW),  the  rating  which  includes  Jezebel  operators,  are  a 
minimum  General  Classification  Test  (GCT)  and  arithmetic  test  (ARI) 
combination  of  110,  successful  completion  of  an  aviation  physical 
examination  and  the  applicant  must  volunteer  for  AW  training 
(approximately  7%  of  the  applicants  are  selected  for  training) . 
There  are  less  than  2,100  AWs  on  active  duty  in  the  Navy  and  of  these 
about  half  are  designated  Jezebel  Operators.   The  training  that 
leads  to  assignment  to  a  flight  crew  in  a  squadron  takes  nearly  nine 
months  and  is  in  the  top  25%  of  Navy  schools  on  the  basis  of  the 
cost  of  training.  After  assignment  to  a  squadron,  training  continues 
until  designation  as  a  member  of  an  Alfa  Crew  which  indicates  achieve- 
ment of  a  high  level  of  operational  readiness  and  completion  of  a 
squadron  training  syllabus.  Acquisition  of  basic  skills,  maintenance 
of  skill,  and  operational  flying  are  a  full  time  task.   Because  of 
this  AWs  are  becoming  more  and  more  an  autonomous  group,  freed  from 


This  information  was  made  available  by  the  Bureau  of  Personnel 
rating  detailer. 
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most  other  duties  except  watch  standing  and  certain  military 
obligations.   They  are  not  trained  in  a  secondary  technical 
specialty. 
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IV.   PRESENTATION  OF  DATA 

In  the  instructions  for  the  raters  they  were  asked  to  circle 
the  number  that  best  fit  the  man's  behavior  in  the  area  described 
by  the  item.  They  were  further  instructed  to  complete  one  item  for 
all  men,  then  to  proceed  to  the  next  item  and  to  complete  it  for  all 
of  the  men. 

In  preparing  the  data  for  analysis  the  score  on  the  items  was 
totaled  to  give  an  overall  grade.   In  some  of  the  analyses  the 
total  score  was  used  and  in  other  sections  the  score  on  a  particular 
item  or  an  average  score  was  used.  Averaging  scores  was  possible 
since  each  man  was  rated  by  five  independent  raters . 

The  scores  that  were  recorded  are  included  in  the  data 
section. 

A.   INTERRATER  RELIABILITY  (IRR) 

The  first  analysis  was  a  test  for  the  interrater  reliability 
of  the  form  using  the  Pearson  Product-Moment  Correlation: 


_  NEXY  -  (EX) (EY) 


/[NEX^  -  (EX)^][NEY^  -  (EY)2] 


where: 

r  is  the  correlation 

X  is  the  value  of  one  variable 

Y  is  the  value  of  the  other  variable 

N  is  the  number  of  cases 
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The  results  of  this  analysis  are  listed  in  Table  I.   The  data  from 
which  the  correlations  were  computed  are  included  as  Appendix  C. 

The  statistical  significance  was  computed  for  each  correlation 
using  the  "Student's  t": 


(n-2) 
t  =  r  - v 


(1-r2) 


where: 

r  is  the  correlation. 

n  is  the  number  of  cases. 

The  second  statistical  test  was  a  comparison  of  the  interrater 
reliability  based  on  the  rank  ordering  that  the  raters  were  asked 
to  do  after  completing  the  form  and  the  rank  ordering  generated  from 
the  scores  taken  from  the  rating  forms.   For  this  comparison  the 
Spearman  Rank  Order  Correlation  rho  (p)  was  computed.   Both  sets  of 
correlations  are  included  in  Table  II  for  ease  of  comparison.   The 
formula  for  computing  the  Spearman  rho  is: 


6ED2 
p  =  1  -  ■ 


N(N2  -  1) 


where: 

D  is  the  difference  in  rank  ordering  for  the  two  criteria 
being  compared. 

N  is  the  number  of  elements  that  have  been  ordered. 
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TABLE  I 


Interrater  Reliability- 


Correlation  . 

(between  raters)  n 

rlj2  =  .639*  13 

r,  =  .545*  13 

r.  .  =  .074  13 
1,4 

r  =  .617*  14 

r,  ,  =  .824**  13 

r2j4  =  .386  13 

r  =  931**  14 

r2,5  *y,5i 

r.  .  =  .544*  13 
3,4 

r  =  .916**  14 

r.  _  =  .485*  14 
4,5 


Note.-- 

The  subscript  indicates  the  specific  rater 
involved,  i.e.,  r.  -  is  the  correlation  between  the 

ratings  of  rater  1  and  rater  2. 

n  =  14  whenever  rater  5  is  involved  because  he 
was  the  division  CPO,  familiar  with  each  of  the  Jezebel 
Operators  but  not  qualified  himself  and,  as  such,  not 
included  in  the  sample.   In  all  other  cases  the  two 
raters  being  compared  were  excluded  from  the  sample. 

*Significant  at  the  .05  level. 
**Significant  at  the  .01  level. 
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TABLE  II 


Interrater  Comparison  Table 


(Ran* 

IRR 
;  Ordering) 

IRR 
(Form) 

"1,2 

= 

.639** 

.761** 

"1,3 

= 

.095 

.586* 

"1,4 

= 

.100 

.01 

"1,5 

= 

.266 

.746** 

"2,3 

= 

.56* 

.745** 

"2,4 

= 

0 

.195 

"2,5 

= 

.604* 

.852** 

"3,4 

= 

.123 

.255 

P3,5 

= 

.600* 

.862** 

"4,5 

= 

.139 

.313 

n 


13 
13 
13 
14 
13 
13 
14 
13 
14 
14 


*Significant  at  the  .05  level. 
**Significant  at  the  .01  level. 
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As  can  be  seen  from  Column  1  of  Table  II,  there  was  only 
occasional  agreement  in  rank  ordering,  however  quite  good 
agreement  was  achieved  using  the  form.  There  was  much  better 
agreement  among  the  raters  when  the  form  was  used  than  when  a 
global  rank  ordering  was  made.   The  reasons  that  the  one  rater, 
rater  number  four,  tended  to  differ  from  the  others  can  only  be 
generalized  from  the  situation  and  from  other  studies.   (It  is 
interesting  to  note  that  he  was  rated  highly  in  motivation  by  his 
peers  and  he  expressed  very  positive  feelings  about  the  form.) 
Perhaps  the  most  likely  explanation  stems  from  the  type  of  training 
that  is  done  by  the  squadrons.  After  an  operator  completes  the 
formal  Navy  school  and  is  assigned  to  a  squadron  the  remainder  of 
his  training  is  conducted  by  the  senior  operators  in  the  squadron. 
When  a  man  is  assigned  to  a  crew  his  sphere  of  contacts  is  reduced 
significantly.   The  result  of  all  this  (coupled  with  social  inter- 
action) is  an  unequal  familiarity  among  the  operators.   Many  studies 
have  been  done  on  the  reliability  of  human  judges.  The  limitations 
and  biasses  are  well  known  and  include  such  phenomenon  as  "halo 
error,"  "leniency  error,"  "similarity  error,"  and  others  (see,  for 
example,  Kelly,  1967).  The  situation  that  exists  in  a  typical 
squadron  could  easily  amplify  these  errors  and  could  distort  or 
invert  the  relative  rating  given  to  some  of  the  men  by  one  of  the 
raters.   This  is  considered  to  be  the  most  probable  explanation 
for  the  one  differing  rater. 
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B.   ITEM  ANALYSIS 

The  analysis  of  the  responses  to  the  individual  items  was 
undertaken  using  four  general  methods.   First,  the  means  and 
standard  deviations  for  all  responses  to  each  item  by  each  rater 
were  computed,  this  was  then  broken  down  into  two  groups;  one  by 
the  military  rank  of  the  person  being  rated  and  another  according 
to  the  rater.   Then  a  correlation  of  the  average  score  on  an  item 
with  the  man's  total  score  was  calculated.   Table  III  contains  the 
composite  score,  the  breakdown  by  seniority  and  the  correlation  with 
total  score.   Table  IV  is  a  listing  of  the  means  for  each  rater. 

Table  III  can  be  used  to  determine  the  items  that  contribute 
most  to  the  total  score  and  discriminate  well  among  the  men.   If 
an  item  describes  a  degree  of  behavior  that  is  observed  or  expected 
in  all  men  then  it  should  have  a  high  mean  and  a  low  variance  and 
will  not  have  the  ability  to  discriminate  between  men.  Therefore,  a 
mean  near  the  midpoint  is  desirable  as  is  a  variance  at  least  large 
enough  to  indicate  a  spread  of  scores.   Another  desirable  character- 
istic for  an  item  is  a  high  correlation  with  total  score.  During  the 
interviews  conducted  immediately  after  the  forms  were  completed  it 
became  obvious  that  the  importance  of  one  dimension  had  been  over- 
looked; that  was  the  differences  in  behavior  attributable  to 
different  levels  of  experience.   As  an  example  of  this  difference, 
a  motivated  man  undergoing  training  as  an  AW  might  show  his 
motivation  by  spending  extra  time  in  the  flight  simulators  or 
trainers,  whereas  an  operator  who  had  achieved  a  high  level  of 
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TABLE  III 
Item  Comparison  Table 


Item 

Mean 
All 

Var. 
All 

Mean 
Senior 

Var. 
Senior 

Mean 
Junior 

Var. 
Junior 

Corre- 
lation 
All 

A 

3.65 

1.74 

4.00 

1.53 

3.31 

1.75 

.832 

B 

4.19 

2.10 

4.97 

0.92 

3.41 

2.09 

.972 

C 

3.69 

1.72 

4.03 

1.24 

3.36 

2.03 

.737 

D 

4.27 

2.17 

5.05 

0.73 

3.49 

2.41 

.942 

E 

4.27 

2.49 

4.87 

1.80 

3.66 

2.50 

.832 

F 

4.29 

2.13  ' 

5.00 

1.10 

3.59 

2.10 

.876 

G 

2.36 

4.21 

3.87 

2.38 

0.84 

1.45 

.920 

H 

4.09 

2.11 

4.67 

1.07 

3.51 

2.52 

.679 

I 

2.63 

3.80 

3.55 

2.31 

1.53 

3.42 

.923 

J 

3.29 

2.50 

4.17 

1.79 

2.38 

1.64 

.880 

K 

2.83 

4.17 

3.74 

2.50 

1.88 

4.24 

.869 

L 

2.60 

3.13 

2.72 

3.37 

2.49 

2.94 

.621 

M 

3.83 

2.61 

4.43 

1.86 

3.15 

2.63 

.921 

N 

3.99 

1.99 

4.50 

0.94 

3.46 

2.55 

.884 

0 

3.45 

2.40 

3.91 

1.26 

2.94 

3.22 

.780 

P 

4.05 

2.13 

4.67 

1.54 

3.39 

1.96 

.955 

Correlation  of  the  item  average  by  all  raters  with  total 
scores  by  all  raters. 
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TABLE  IV 


Rater  Comparison  Table 


Difference 

Rater 

Between 
Means 
(H-L) 

Item 

1 

2 

3 

4 

5 

A 

Mean 

3.40 

4.25 

3.73 

3.87 

3.00 

1.25 

Var. 

2.83 

1.80 

0.64 

1.05 

1.87 

B 

Mean 

4.13 

4.81 

3.73 

4.25 

4.00 

1.12 

Var. 

2.12 

1.76 

1.35 

2.47 

2.67 

C 

Mean 

4.80 

3.81 

3.47 

3.19 

3.25 

1.61 

Var. 

1.31 

1.63 

0.84 

2.16 

1.27 

D 

Mean 

5.20 

4.69 

4.27 

3.56 

3.69 

1.60 

Var. 

2.46 

1.69 

0.78 

2.26 

2.23 

E 

Mean 

4.93 

4.37 

5.20 

2.44 

4.56 

2.76 

Var. 

1.30 

2.12 

1.31 

1.99 

1.20 

F 

Mean 

3.67 

4.63 

5.00 

3.69 

4.50 

1.33 

Var. 

2.24 

2.92 

1.14 

2.10 

1.33 

G 

Mean 

2.80 

2.13 

2.07 

2.69 

2.13 

0.73 

Var. 

4.31 

4.52 

2.21 

4.50 

5.98 

H 

Mean 

4.67 

4.19 

3.20 

3.94 

4.44 

1.47 

Var. 

2.81 

2.96 

1.03 

1.80 

1.20 

I 

Mean 

4.60 

2.94 

2.43 

1.75 

1.44 

3.16 

Var. 

1.97 

3.53 

1.29 

1.13 

4.13 

J 

Mean 

2.87 

4.00 

3.00 

3.06 

3.31 

1.13 

Var. 

2.12 

2.27 

3.20 

2.20 

3.01 

K 

Mean 

5.13 

2.19 

2.67 

2.56 

1.63 

3.50 

Var. 

1.55 

1.63 



2.86 

3.98 

L 

Mean 

3.80 

3.69 

2.47 

3.06 

0.06 

3.74 

Var. 

2.89 

1.56 

1.27 

0.73 

0.06 

(Table  continued  on  next  page) 
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TABLE  IV  (continued) 


Rater 

Difference 
Between 
Means 

Item 

1 

2 

3 

4 

5 

(H-L) 

M 

Mean 
Var. 

3.60 
3.11 

4.06 
3.53 

3.71 
1.24 

3.75 
3.00 

3.94 
1.93 

0.46 

N 

Mean 
Var. 

4.67 
1.67 

4.13 
2.92 

3.75 
0.50 

3.81 
2.70 

3.50 
1.07 

1.17 

0 

Mean 
Var. 

4.8 
1.46 

3.06 
1.26 

3.75 
0.92 

3.69 
2.76 

2.25 
1.40 

2.55 

P 

Mean 
Var. 

4.46 
1.98 

4.63 
1.58 

4.42 
0.81 

3.56 
2.40 

3.31 
2.65 

1.37 

Avg. 

Mean 

4.22 

3.85 

3.32 

3.30 

3.06 

competence  would  be  unlikely  to  request  any  more  trainer  periods  than 
the  minimum  monthly  requirement.  As  a  correlate,  a  motivated  and 
experienced  man  might  demonstrate  his  motivation  by  working  to  qualify 
in  other  crew  positions,  an  activity  that  would  not  be  expected  of  a 
junior  man  working  toward  his  initial  qualification.  This  differ- 
entiation by  experience  level  existed  in  some  areas,  however  it  was 
not  a  characteristic  of  all  items,  many  of  the  items  were  equally 
applicable  to  junior  and  experienced  men. 
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The  method  used  to  evaluate  items  for  their  applicability  to 
a  particular  experience  group  was  to  use  the  difference  in  the  means 
of  the  two  groups  to  identify  items  with  a  likelihood  of  having  this 
characteristic.  An  item  with  a  relatively  high  difference  should 
favor  the  experienced  group  and  a  low  difference  should  indicate  an 
item  of  greater  applicability  to  the  lower  experience  level.   (In  no 
case  was  the  mean  of  the  senior  operators  lower  than  the  mean  of  the 
junior  operators.   The  average  difference  in  the  mean  was  1.4  so  that 
smaller  differences  tend  to  identify  items  more  characteristic  of 
junior  men. ) 

Another  characteristic  of  a  desirable  item  is  that  it  should  be 
marked  in  a  similar  range  by  all  raters  when  rating  the  same  group. 
Table  IV  shows  the  mean  score  on  each  item  by  each  rater  and  the 
difference  between  the  high  and  low  score.  A  large  difference  tends  to 
identify  items  that  have  been  interpreted  differently  by  different 
raters.  An  exception  to  this  logic  occurs  when  one  rater  has  a  mean 
which  is  much  different  from  the  others  since  either  a  high  or  low  mean 
acts  to  limit  the  possible  scores  in  one  tail  while  increasing  the 
possible  scores  in  the  other  tail.   In  this  case  a  slight  change  in 
the  wording  may  greatly  improve  the  suitability  of  the  item.  One 
rater,  the  rater  whose  interrater  reliability  was  low,  tended  to  differ 
markedly  from  the  others  at  least  twice  as  often  as  did  any  other  rater. 

To  illustrate  how  some  of  these  desirable  characteristics  have  been 
applied  to  the  items,  compare  the  figures  in  the  last  column  of  Table  IV, 
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Three  of  the  items  have  differences  between  means  greater  than  3.00 
(items  I,  K,  and  L)  .  On  examination  of  these  items  it  seems  logical 
to  assume  that  the  interpretation  could  be  different  for  the  different 
raters.   Probably  item  I  is  the  item  most  susceptible  to  "extension 
into  the  future."  That  is,  evaluating  the  phrase  in  the  instructions, 
"if  the  circumstances  were  appropriate,"  to  include  situations 
considerably  beyond  that  which  was  expected  or  interpreted  by  most 
raters . 

The  wording  of  item  K  is  such  that  it  does  not  adequately  describe 
or  differentiate  a  highly  motivated  man.  As  it  is  written  it  could  be 
broadly  interpreted  to  include  the  majority  of  sailors  and  this 
apparently  was  the  interpretation  made  by  some  of  the  raters.   Item  L 
was  very  much  subject  to  the  experience  element  described  earlier. 
Limiting  this  item  to  evaluation  for  a  particular  experience  group 
should  ease  the  problem  of  interpretation. 

From  this  analysis  a  new  rating  form  was  developed  and  is  included 
as  Appendix  B.   This  form  should  have  greater  interrater  reliability, 
better  discrimination  between  highly  motivated  junior  men  and  the 
senior  men  whose  motivation  is  marginal  and  it  should  be  even  more 
acceptable  to  the  users.   In  preparing  this  form,  items  with  a  narrow 
variance  or  a  very  high  or  low  mean  were  either  modified  or  eliminated 
and  the  information  from  Table  IV,  together  with  a  "face"  determination 
was  used  to  separate  items  by  their  applicability  to  a  particular 
experience  group. 
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C.  POLICY  CAPTURING 

Certain  exploratory  attempts  were  made  to  understand  the  criteria 
the  raters  used  in  the  free  rank-ordering.   These  efforts  at  policy 
capturing  were  rudimentary  and  not  originally  planned  for  inclusion 
in  this  study.   The  working  concept  was  that  in  general  raters  tend 
to  make  evaluations  based  on  a  limited  number  of  elements  even  if  much 
more  data  is  available  (Whisler  §  Harper,  1962).   Because  of  this  it 
seemed  possible  that  the  rank-ordering  was  made  on  the  basis  of  only 
a  few  types  of  behavior  and,  if  this  was  the  case,  they  should  be 
among  the  incidents  described  in  the  items  on  the  form.  To  test  for 
any  primary  relationship  a  scatter  plot  was  drawn  for  each  item  showing 
the  average  score  that  the  man  was  given  by  the  five  raters  on  the  item, 
plotted  against  his  average  rank  order.  This  was  largely  unsuccessful, 
although  two  items  (0  and  B)  did  have  a  plot  somewhat  similar  to  the 
expected  shape.   If  a  strong  correlation  had  been  found,  it  might  have 
pointed  to  a  simplification  of  the  rating  technique  or  of  the  definition 
of  motivation. 

This  analysis  was  undertaken  to  determine  if  further,  more  exhaustive 
techniques  such  as  factor  analysis  were  warranted.   Because  of  the  weak 
correlation  and  the  limited  data  no  further  investigation  was  undertaken 
at  this  time. 

D.  PROBLEMS  AND  UNANSWERED  QUESTIONS 

In  the  conduct  of  the  project  several  problems  came  up  which  deserve 
discussion.   First  was  the  question  of  the  character  of  motivation  in 
the  population  sampled.   Does  the  full  spectrum  exist  from  highly 
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favorable  motivation  to  strongly  negative  motivation,  or  is  the  range 
limited  (because  of  the  selection  procedures)  to  the  area  from  neutral 
to  highly  favorable  motivation?  During  the  interviews  it  was  impossible 
to  generate  neutral  or  negative  incidents.   It  is  difficult  to 
conceptualize  examples  of  low  or  negative  motivation  other  than 
apathy,  disinterest,  or,  in  the  extreme,  destructiveness .   This  may 
be,  in  part,  related  to  the  questioning  or  interviewing  technique 
used  where  positive  or  favorable  situations  were  set  up  initially. 
If  the  procedure  were  reversed  it  is  possible  that  neutral  or  negative 
incidents  could  have  b.een  developed. 

One  problem  that  is  frequently  perplexing  in  the  analysis  of 
evaluations  is  the  so-called  "halo  effect,"  the  propensity  for  raters 
who  know  a  man  who  should  be  rated  high  in  one  area  (for  example, 
promptness)  to  rate  him  high  in  other  areas  for  which  they  do  not 
have  an  adequate  basis  for  evaluation.   It  can  be  expected  that  this 
would  occur  on  a  motivation  evaluation.  However,  what  may  appear  to 
be  "halo  error"  is  expected  if  each  item  is  really  measuring  the  same 
thing.  Secondly,  such  an  error  may  not  be  detrimental  to  the  ultimate 
result  in  the  early  stages  of  development  if  certain  necessary  behavioral 
descriptions  have  not  been  included.   This  possibility  should  be 
alleviated  when  and  if  motivation  is  broken  down  to  dimensions  which 
can  be  put  on  scales.  While  motivation  is  very  likely  multidimensional 
the  detailed  investigation  of  these  dimensions  might  best  proceed  from 
an  initially  global  definition  or  treatment. 
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A  clear  and  present  problem  in  the  measurement  of  any  complex 
characteristic  is  the  validity  of  the  method  chosen.  This  problem 
is  a  continuing  one  and  one  which  can  only  be  decreased  with  extensive 
and  lengthy  techniques.   Further,  it  is  intensified  in  the  early  stages 
of  test  development.   There  are  at  least  two  types  of  validity  which 
the  Critical  Incident  Techniques  insure.   First  is  what  might  be 
termed  source  validity;  it  is  a  valid  measure  of  motivation  because 
it  was  developed  from  descriptions  of  motivated  behavior.   To  a 
degree  this  same  fact  can  be  considered  to  imply  consenual  validity 
and  the  consensus  was  further  guaranteed  by  the  post-test  interviews. 
Face  validity  is  at  best  an  insufficient  test,  however,  the  form  as 
administered  certainly  does  have  face  validity. 

To  improve  the  validity,  to  generate  a  numerical  value  for  the 
validity  of  this  instrument  for  measuring  motivation  will  pose  certain 
problems.   First  and  most  imposing  is  the  problem  that  arises  from  the 
fact  that  this  investigation  was  at  the  limit  of  current  understanding 
of  motivation.  Because  of  this  there  are  no  other  equal  or  similar 
intermediate  criterion  against  which  this  can  be  compared.  However, 
certain  schemes  can  be  envisioned  which  might  involve  setting  some 
situation  or  series  of  situations  in  which  a  motivated  man  might  be 
expected  to  perform  in  a  particular  fashion.   These  concepts  are  only 
fragmentary  and  conjectural  and  would  require  a  great  deal  of  additional 
thought  and  testing  before  they  could  be  attempted. 
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V.   CONCLUSIONS  AND  RECOMMENDATIONS 


The  most  significant  conclusion  that  can  be  drawn  from  this  study- 
is  that  an  instrument  was  developed  that  improved  the  reliability  of 
ratings  of  a  person's  motivation.  Since  this  work  was  in  an  area  that 
has  been  virtually  untouched,  it  was  impossible  to  relate  the  results 
to  any  other  findings  for  comparison. 

A  second  conclusion  that  seems  to  be  warranted  is  that  such  an 
instrument  would  be  applicable  across  a  wide  range  of  different  skills 
and  tasks  with  only  minor  modification.  This  conclusion  was  supported 
by  the  very  general  nature  of  the  items  that  were  generated  in  the 
interviews  even  though  special  attention  was  given  to  elicit  and 
report  incidents  with  the  highest  possible  degree  of  specificity. 
This  characteristic,  while  not  completely  unexpected,  was  encountered 
to  a  surprising  extent.   It  is  considered  that  a  very  few  different 
forms  could  be  developed  to  cover  all  but  the  most  extraordinary  of 
occupations  or  individuals.   This  conclusion  will  require  additional 
study  to  verify. 

It  is  certainly  not  expected  that  a  strong  relationship  exists 
between  task  or  job  motivation  and  career  motivation.  However,  it  does 
seem  likely  that  a  low  task  motivation  should  help  to  identify  people 
who  would  be  less  desirable  to  retain  in  the  military. 

There  is  a  need  to  look  at  the  variation  of  motivation  over  time 
and  to  test  this  variation  with  performance.  The  time  dimension  is 
one  which  has  not  been  investigated. 
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The  low  relationship  between  morale  or  job  satisfaction  and 
production  is  well  known,  however,  it  seems  possible  that  there  may 
be  a  more  complex  interrelation  with  motivation  being  a  key  element. 
This  possibility  can  be  investigated  through  a  motivation  form. 

One  of  the  most  intriguing  possibilities  is  the  use  of  these 
findings  and  the  methods  that  were  used  to  discover  and  to 
appropriately  value  incentives.  The  relationship  between  the 
behavior  that  was  reported  as  demonstrative  of  motivation  and 
incentives  that  might  be  useful  in  motivating  workers  may  at  first 
seem  remote.  However,  many  times  during  the  interviews  the  situation 
was  an  important  element  in  the  display  of  motivated  behavior.   It 
was  often  the  case  that  the  situation  seemed  to  be  the  motivating 
factor  and  as  such  should  be  viewed  as  an  important  incentive. 
These  descriptions  were  strongly  supportive  of  Herzberg's  "Two 
Factor  Theory,"  the  most  profound  motivation  came  as  a  direct  result 
of  doing  the  job  the  men  are  trained  to  do.  During  the  interviews 
it  became  obvious  that  problems  in  communicating  ideas  and  value 
systems  from  the  lowest  levels  in  an  organization  to  levels  where 
meaningful  action  can  be  taken  are  still  very  much  the  rule.  Attempts 
at  uncovering  valued  incentives  should  help  to  bridge  this  gap. 

Most  personnel  evaluation  systems  require  a  rating  of  personal 
attributes  or  personality  variables  and  the  value  of  these  variables 
to  an  organization  is  highly  subjective  at  best.   Personality  traits 
are  extensively  used  in  evaluating  the  performance  of  people  who  are 
far  removed  from  a  direct  influence  on  production.   It  may  well  be  that 
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motivation  is  a  trait  that  is  meaningful  to  an  organization  and  can 
be  measured  more  precisely,  reliably  and  more  objectively  than  many 
of  the  traits  currently  being  rated.   There  is  the  added  possibility 
that  it  will  have  greater  correlation  with  relevant  performance  than 
the  other  traits . 

The  development  of  a  well  tested,  reliable  and  accurate  measurement 
technique  for  motivation  and  one  that  is  relatively  free  from  the 
effects  of  performance  would  provide  a  powerful  new  tool  for  further 
investigation  of  numerous  concepts  of  management  policy  and  leadership 
techniques.   It  is  possible  that  changes  in  motivation  might  be  a  much 
more  valid  measure  of  leadership  and  management  ability  than  output  or 
performance  in  fields  where  ultimate  criteria  can  only  be  inferred  or 
that  are  strongly  influenced  by  equipment  differences  or  situational 
aspects  which  are  not  easily  assessed. 

Certainly  this  project  has  done  one  thing  and  that  is  to  generate 
an  objective  operational  definition  of  task-oriented  motivation. 
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APPENDIX  A 


MOTIVATION 


Background  Information:   This  evaluation  form  is  an  attempt  to  develop 
an  objective  measure  of  motivation.  The  items  that  are  included  have 
been  taken  from  descriptions  of  highly  motivated  men. 

Directions:   Circle  the  number  that  best  indicates  how  characteristic 
the  item  is  of  the  man  being  rated.   The  evaluation  should  be  based 
upon  direct  observation  wherever  possible.  When  direct  observation 
has  not  occurred,  indicate  whether  you  think  the  man  would  act  in  the 
manner  described  if  the  circumstances  were  appropriate.  This  evalu- 
ation is  directed  at  mission  or  job-oriented  motivation  only. 


Item 


Frequently 
6   5   4 


A.  Asks  probing  important  questions; 
questions  that  are  difficult  to  ans- 
wer and  often  require  special  effort 
or  research.  6 


Occasionally   Never 


3   2   1 


3   2   1 


0 


B.  Doesn't  wait  to  be  told  to  do 
something,  sees  things  that  need 
to  be  done  and  does  them. 


5   4 


3   2   1 


C.  Seeks  out  other  operators  that 
you  consider  to  be  motivated,  enjoys 
being  around  professionally  competent 

men.  Avoids  poorly  motivated  men.    6   5   4     3   2   1 

D.  Has  taken  time  out  to  help  some- 
one with  a  problem.  6   5   4     3   2   1 

E.  Discusses  advancement  and  profi- 
ciency exams  with  other  operators  at 

length  shortly  after  taking  them.     6   5   4     3   2   1 

F.  Watches  and  assists  mechanics 
or  technicians  in  the  repair  of 
equipment  to  help  them  and  to 

improve  his  own  knowledge.  6   5   4     3   2   1 
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Item 


Frequently 
6   5   4 


Occasionally   Never 
3   2   1       0 


G.  Has  given  lectures  ans  has  pre- 
pared lesson  plans  when  he  was  not 
required  to  do  so. 

H.  Has  gone  to  extra  effort  to  get 
information  or  to  prepare  himself 
ahead  of  time  to  help  make  a  flight 
go  smoothly. 

I.  Has  originated  improvements  in 
equipment  or  procedures . 

J.  Has  shown  interest  in  the  other 
crew  positions  in  the  aircraft 
and  has  taken  positive  steps  to 
qualify  in  one  or  more  of  them. 

K.  Has  come  back  to  the 
squadron  from  TAD  or  even  from 
leave  to  attend  an  important 
lecture  or  to  complete  an 
important  task. 

L.   Spends  more  than  the 
minimum  time  in  the  trainers . 
Asks  for  additional  trainer 
periods . 

M.   Has  contacted  other 
squadrons,  the  Wing,  the  ASCAC 
or  FAETUPAC  to  get  any  new  or 
relevant  information. 

N.  Was  reluctant  or  hesitant  to 
leave  an  important  or  interesting 
situation  directly  related  to  the 
ASW  mission. 

0.  Starts  the  pref light  checks  of 
equipment  prior  to  the  normal  time 
in  order  that  any  problems  encoun- 
tered can  be  corrected. 

P.  Has  stayed  on  a  job  until  it 
was  finished  regardless  of  regular 
working  hours . 
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The  following  items  can  be  answered  by  either  Yes  or  No. 


Item 

Q.  Has  learned  a  special  procedure  on  his  own 

from  manuals  and  publications  and  is  considered 

to  be  an  expert  in  this  area  by  the  personnel  in 

the  squadron  and,  perhaps  in  other  squadrons  as 

well.  Yes      No 

R.   Is  interested  in  making  sure  that  important 

intelligence  information  is  sent  to  higher 

levels  in  the  information  chain.  Yes      No 

S.  Has  an  eagerness  to  find  and  share  new  information 

with  other  squadron  personnel .  Yes      No 

T.  Assumes  that  the  information  that  is  being 

developed  on  a  mission  is  important  and  will  be  used 

later  on  by  someone  who  wasn't  there  when  it  was 

collected;  keeps  his  logs  and  grams  accordingly.  Yes      No 


Name  of  man  being  rated 
Rater's  initials 
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APPENDIX  B 


MOTIVATION 
(REVISED) 

Background  Information:   This  evaluation  form  is  an  attempt  to  develop 
an  objective  measure  of  motivation.  The  items  that  are  included  have 
been  taken  from  descriptions  of  highly  motivated  men. 

Directions:   Circle  the  number  that  best  indicates  how  characteristic 
the  item  is  of  the  man  being  rated.   The  evaluation  should  be  based 
upon  direct  observation  wherever  possible.  When  direct  observation 
has  not  occurred,  indicate  whether  you  think  the  man  would  act  in  the 
manner  described  if  the  circumstances  were  appropriate. 

The  first  five  items  (A-E)  should  be  completed  for  junior  men  only 
Junior  in  this  case  should  be  considered  as  not  having  achieved  Alfa- 
Crew  status. 


Item 


Increasingly  Favorable 


A.  Asks  probing,  important  questions; 
questions  that  are  difficult  to  answer 
and  often  require  special  effort  or 
research. 

B.  Spends  more  than  the  minimum  time 
in  the  trainers.  Asks  for  additional 
trainer  periods . 

C.  Seeks  out  other  operators  that 
you  consider  to  be  motivated,  enjoys 
being  around  professionally  competent 
men. 

D.  Starts  the  pref light  checks  of 
equipment  prior  to  the  normal  time 
in  order  that  any  problems  encoun- 
tered can  be  corrected. 

E.  Has  gone  to  extra  effort  to  get 
information  or  to  prepare  himself 
ahead  of  time  to  help  make  a  flight 
go  smoothly. 
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The  second  five  items  (F-J)  should  be  completed  for  senior  operators 

Increasingly  Favorable 


Item 


0 


F.  Has  given  lectures  and  has  prepared 
lesson  plans  when  he  was  not  required  to 

do  so.  0   1 

G.  Has  come  back  to  the  squadron  from 
TAD  or  even  from  leave  to  attend  an 
important  task.  0   1 

H.  Has  originated  improvements  in 

equipment  or  procedures .  0   1 

I.  Has  shown  interest  in  the  other 

crew  positions  in  the  aircraft  and 

has  taken  positive  steps  to  qualify 

in  one  or  more  of  them.  0   1 

J.  Has  taken  time  out  to  help  someone 

with  a  problem.  0   1 


3   4   5   6 


3   4   5   6 


3   4   5   6 


3   4   5   6 


3   4   5   6 


The  remaining  items  should  be  completed  for  all  operators. 


K.  Has  contacted  other  squadrons, 
the  Wing,  the  ASCAC  or  FAETUPAC  to 
get  any  new  or  relevant  information. 

L.  Was  reluctant  or  hesitant  to 
leave  an  important  or  interesting 
situation  directly  related  to  the 
ASW  mission. 

M.  Discusses  advancement  and  profi- 
ciency exams  with  other  operators  at 
length  shortly  after  taking  them. 

N.  Has  stayed  on  a  job  until  it  was 
finished  regardless  of  regular  working 
hours . 

0.  Watches  and  assists  mechanics  or 
technicians  in  the  repair  of  equip- 
ment to  help  them  and  to  improve 
his  own  knowledge. 


3   4   5   6 


3   4   5   6 


3   4   5   6 


3   4   5   6 


3   4   5   6 
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Item 


P.  Doesn't  wait  to  be  told  to  do 
something,  sees  things  that  need  to 
be  done  and  does  them. 

Q.   Is  interested  in  making  sure  that 
important  intelligence  information  is 
sent  to  higher  levels  in  the  information 
chain. 

R.  Has  an  eagerness  to  find  and  share 
new  information  with  other  squadron 
personnel. 

S.  Assumes  that  the  information  that 
is  being  developed  on  a  mission  is 
important  and  will  be  used  later  on 
by  someone  who  wasn't  there  when  it 
was  collected;  keeps  his  logs  and 
grams  accordingly. 


Increasingly  Favorable 


0 


Name  of  man  being  rated 
Rater's  initials 
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DATA 

SECTION  3 

Average 
Total 
Score 

Average  Score  on 

Item 

A 

B 

C 

D 

E 

F 

G 

H 

B  69.00 
B*  39.75 
Q     22.80 
C*  69.40 
CZ   54.00 
F  54.40 

4 

5.50 

3.20 

4.40 

4.50 

4.75 

3.75 

4.20 

2.75 

3.30 

3.00 

2.40 

3.50 

4.25 

.50 

3.75 

2.20 

2.00 

2.40 

1.80 

3.20 

1.60 

0.00 

1.40 

4.00 

5.40 

4.80 

5.20 

5.40 

5.40 

4.20 

4.40 

4.40 

4.00 

4.00 

4.40 

3.80 

4.00 

.80 

3.80 

3.40 

4.40 

3.60 

4.20 

4.60 

4.40 

2.60 

4.00 

H  68.00 

3.60 

4.80 

4.40 

5.00 

5.40 

5.20 

4.20 

4.20 

K  50.60 
IC  55.40 
IC  54.20 
L  76.40 

3.40 

4.20 

3.60 

3.40 

4.60 

4.40 

1.60 

4.20 

4.80 

4.00 

4.60 

4.60 

4.50 

2.60 

2.60 

4.60 

3.20 

3.60 

3.40 

4.40 

3.80 

4.20 

2.00 

4.40 

4.60 

5.60 

3.60 

5.60 

5.40 

5.60 

5.20 

5.40 

M  50.00 

3.40 

4.00 

3.40 

4.00 

4.00 

3.80 

1.40 

3.80 

P  29.40 

2.40 

2.00 

2.20 

3.00 

2.20 

2.40 

.60 

2.60 

R  45.80 

3.00 

3.80 

3.40 

3.60 

3.60 

4.00 

.40 

3.80 

T  74.00 

5.20 

5.60 

4.20 

5.60 

4.20 

5.60 

5.40 

2.70 

W  67.20 

4.00 

5.00 

3.80 

5.00 

5.60 

5.40 

3.60 

4.60 

Average 
Total 
Score 

Average  Score  on 

Item 

I 

J 

K 

L 

M 

N 

0 

P 

B  69.00 
B*  39.75 
C     22.80 
C     69.40 
C     54.00 
F  54.40 

3.00 

4.40 

3.00 

2.75 

4.50 

4.50 

3.50 

5.25 

0.75 

2.00 

1.00 

2.00 

3.50 

3.75 

3.75 

2.67 

0.00 

1.00 

0.75 

1.75 

1.00 

2.00 

1.50 

2.25 

3.25 

3.75 

4.80 

2.80 

4.80 

5.00 

4.50 

4.80 

1.75 

3.25 

2.25 

3.00 

3.80 

4.20 

3.60 

4.40 

2.80 

4.00 

2.50 

1.60 

3.40 

3.60 

2.75 

3.60 

H  68.00 

3.80 

3.75 

3.60 

2.20 

4.00 

4.40 

4.20 

5.20 

K  50.60 
IC  55.40 
IC  54.20 
L  76.40 

2.25 

3.00 

2.00 

2.60 

3.00 

3.80 

3.00 

3.40 

1.75 

2.25 

3.75 

4.20 

4.50 

5.25 

4.60 

4.20 

2.20 

3.25 

2.50 

2.20 

3.75 

4.00 

3.25 

4.20 

4.40 

5.20 

4.00 

3.60 

5.60 

4.60 

4.25 

5.60 

M  50.00 

2.25 

4.25 

2.25 

2.60 

4.00 

3.80 

3.00 

3.80 

P   29.40 

1.50 

2.00 

1.50 

1.60 

1.50 

2.25 

1.50 

2.25 

R  45.80 

2.00 

2.25 

2.50 

2.40 

3.75 

3.75 

3.75 

3.40 

T  74.00 

4.20 

5.25 

3.25 

3.20 

5.20 

5.40 

3.80 

5.20 

W  67.20 

3.80 

4.00 

3.75 

3.20 

4.50 

4.75 

4.25 

4.20 
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